build-support: add new hooks #370869

drupol · 2025-01-04T15:03:24Z

Hello everyone,

During the holidays, I spent some time packaging Python packages and noticed that some of them require a valid and mutable home directory to build or function properly.

Currently, the default behavior of the Nix stdenv is to set the HOME environment variable to /homeless-shelter, a read-only directory. While this approach is consistent, it can cause issues for packages that expect a writable home directory.

After performing a treewide search in nixpkgs, I found approximately 500 instances where export HOME is used (rg -c "export HOME" | wc -l) and 300 where export PATH is used (rg -c "export PATH" | wc -l). These repetitive patterns inspired me to propose a solution adhering to the DRY principle.

I created custom hooks, tmpdirAsHomeHook, which sets up a mutable home directory automatically when added to nativeBuildInputs or another appropriate place. Additionally, addBinToPath, which modifies the PATH environment variable to include bin/ if available.

I’d appreciate your feedback on this approach and have a few questions:

Do you think this would be a valuable addition to nixpkgs?
Regarding naming, I’m considering renaming tmpdirAsHomeHook to mutableHomeDirHook for better clarity. What do you think?
In the hooks, I use the line addEnvHooks "$targetOffset" addBinToPath. I’m not entirely confident about the best way to implement this. Could you provide some guidance?

Thanks for taking the time to review this. I look forward to hearing your thoughts and suggestions!

Best regards

Things done

Add a 👍 reaction to pull requests you find important.

nixos-discourse · 2025-01-04T15:09:20Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/request-for-comments-adding-two-new-hooks/58282/1

pkgs/build-support/setup-hooks/writable-tmpdir-as-home.sh

xiaoxiangmoe · 2025-01-13T15:45:55Z

Will this hook also be avaliable in preConfig or some thing like this?

Here is some example:

nixpkgs/pkgs/by-name/af/affine/package.nix

Line 109 in 1a685a9

export HOME="$NIX_BUILD_TOP"

nixpkgs/pkgs/by-name/af/affine/package.nix

Line 156 in 1a685a9

export HOME="$NIX_BUILD_TOP"

drupol · 2025-01-13T15:49:15Z

Will this hook also be avaliable in preConfig or some thing like this?

Here is some example:

nixpkgs/pkgs/by-name/af/affine/package.nix

Line 109 in 1a685a9

export HOME="$NIX_BUILD_TOP"

nixpkgs/pkgs/by-name/af/affine/package.nix

Line 156 in 1a685a9

export HOME="$NIX_BUILD_TOP"

It can actually be added to any phase hooks, so yeah.

hsjobeki

I like the Idea overall.
I also have two feedback Points:

This could be two PRs.
For my taste the amount of documentation is not enough. Especially i wouldn like to know stuff that happens internally, like what value is. $HOME set to and when?

drupol · 2025-01-14T06:40:45Z

This could be two PRs.

While I'm all for splitting PR, make sure that they have a clear scope, I really don't understand why I would need to split this one. The scope is clear, even the commits are correctly split. So I'm sorry but no, I won't split this PR in two.

For my taste the amount of documentation is not enough. Especially i wouldn like to know stuff that happens internally, like what value is. $HOME set to and when?

OK I will improve it.

drupol · 2025-01-26T18:30:53Z

Planning to merge this tomorrow morning, feel free to raise your voice if you believe it is a bad idea.

K900 · 2025-01-26T19:10:08Z

I'd like to see some examples of packages helped by these hooks, at the very least.

drupol · 2025-01-26T19:15:04Z

I'd like to see some examples of packages helped by these hooks, at the very least.

Here's some Github search where it would be used:

addBinToPathHook: https://github.com/search?q=repo%3ANixOS%2Fnixpkgs%20export%20PATH%3D%24out%2Fbin%3A%24PATH&type=code
writableTmpDirAsHomeHook: https://github.com/search?q=repo%3ANixOS%2Fnixpkgs+export+HOME%3D&type=code

K900 · 2025-01-27T08:02:21Z

doc/stdenv/stdenv.chapter.md

@@ -1375,6 +1375,26 @@ This hook only runs when compiling for Linux.

 This sets `SOURCE_DATE_EPOCH` to the modification time of the most recent file.

+### `add-bin-to-path.sh` {#add-bin-to-path.sh}
+
+This setup hook adds the `./bin/` directory to the `PATH` environment variable,


This should probably at least explicitly mention it's adding $out/bin and not $sourceRoot/bin or whatever. Honestly, this is my big concern with this hook: it's not immediately obvious what exactly it does, and the thing it does comes with a bunch of weird caveats (what if there are multiple outputs? what if the binaries are in sbin? etc).

Thank you, I just updated the documentation. Let me know if this resolves this feedback.

SuperSandro2000 · 2025-01-31T17:21:25Z

pkgs/build-support/setup-hooks/add-bin-to-path.sh

+
+addBinToPath () {
+    # shellcheck disable=SC2154
+    if [ -d "$out/bin" ]; then


Shouldn't we log something if the directory doesn't exist?

This has been changed since then: https://github.com/NixOS/nixpkgs/pull/378110/files#diff-d390481be231cb8f7d3eaf4cbc0cd6580b5e469d9c3ac6c5a91a8c9fa62f3799

It hasn't been suggested during the month this PR has been opened, so no.
That said, while implementing that hook this morning, I had an issue because of that if condition. I removed it in 62d4ca6

SuperSandro2000 · 2025-01-31T17:22:53Z

pkgs/build-support/setup-hooks/writable-tmpdir-as-home.sh

+
+export HOME
+
+writableTmpDirAsHome () {


If we name the hook "writable" shouldn't it also call chmod +w or so to make sure it is writable? Right now with the wrong umask it isn't

It wasn't needed so far, but it could be done in a future PR.

SuperSandro2000 · 2025-01-31T17:23:07Z

pkgs/build-support/setup-hooks/writable-tmpdir-as-home.sh

+
+writableTmpDirAsHome () {
+    if [ ! -w "$HOME" ]; then
+        HOME="$NIX_BUILD_TOP/.home"


Why is that a hidden directory and not just home?

It has been suggested to name it as such, I don't really have any preference.

nixpkgs-ci · 2025-02-07T12:03:34Z

Backport failed for release-24.11, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-24.11
git worktree add -d .worktree/backport-370869-to-release-24.11 origin/release-24.11
cd .worktree/backport-370869-to-release-24.11
git switch --create backport-370869-to-release-24.11
git cherry-pick -x 87521c59b636ae9073c58bd549b4cdc334a3dc28 f8110737aeedf8ba92ea46a0b1d2d298843d7c06

bjornfor · 2025-02-08T20:07:27Z

pkgs/build-support/setup-hooks/writable-tmpdir-as-home.sh

+
+writableTmpDirAsHome () {
+    if [ ! -w "$HOME" ]; then
+        HOME="$NIX_BUILD_TOP/.home"


I worry that this will break when run under nix-shell. Can we use something other than $NIX_BUILD_TOP please? (Or define a new env var for where the build actually happens, which is also valid in nix-shell?)

Well, break nix-shell and use of the keepBuildTree setup hook, as per the linked issue above.

drupol changed the title ~~treewide: add two custom hooks~~ build-support: add two custom hooks Jan 4, 2025

github-actions bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jan 4, 2025

drupol mentioned this pull request Jan 9, 2025

magento-cloud: init at 1.46.1 #372239

Merged

13 tasks

drupol changed the title ~~build-support: add two custom hooks~~ build-support: add new hook mutableHomeDirHook Jan 9, 2025

drupol changed the title ~~build-support: add new hook mutableHomeDirHook~~ build-support: add new hook tmpdirAsHome Jan 9, 2025

drupol force-pushed the push-zyrxtzozkqnt branch 3 times, most recently from 9e45c4c to c30ae4a Compare January 11, 2025 21:12

github-actions bot added the 8.has: documentation This PR adds or changes documentation label Jan 11, 2025

drupol force-pushed the push-zyrxtzozkqnt branch from c30ae4a to 1a6ea00 Compare January 11, 2025 21:26

drupol marked this pull request as ready for review January 11, 2025 21:30

nix-owners bot requested review from hsjobeki, Ericson2314, GetPsyched and fricklerhandwerk January 11, 2025 21:31

drupol changed the title ~~build-support: add new hook tmpdirAsHome~~ build-support: add new hooks Jan 11, 2025

xiaoxiangmoe reviewed Jan 13, 2025

View reviewed changes

pkgs/build-support/setup-hooks/writable-tmpdir-as-home.sh Outdated Show resolved Hide resolved

drupol force-pushed the push-zyrxtzozkqnt branch from 1a6ea00 to 48ddab5 Compare January 13, 2025 14:52

xiaoxiangmoe reviewed Jan 13, 2025

View reviewed changes

pkgs/build-support/setup-hooks/writable-tmpdir-as-home.sh Outdated Show resolved Hide resolved

drupol force-pushed the push-zyrxtzozkqnt branch from 48ddab5 to 2246ddf Compare January 13, 2025 15:27

xiaoxiangmoe reviewed Jan 13, 2025

View reviewed changes

pkgs/build-support/setup-hooks/writable-tmpdir-as-home.sh Outdated Show resolved Hide resolved

drupol force-pushed the push-zyrxtzozkqnt branch from 2246ddf to e45fb8f Compare January 13, 2025 15:32

hsjobeki reviewed Jan 13, 2025

View reviewed changes

drupol force-pushed the push-zyrxtzozkqnt branch from e45fb8f to e8e0a8d Compare January 14, 2025 07:27

drupol mentioned this pull request Jan 26, 2025

multiqc: init at 1.26 #372510

Merged

12 tasks

drupol force-pushed the push-zyrxtzozkqnt branch 2 times, most recently from 4823191 to 881d72c Compare January 26, 2025 18:20

K900 reviewed Jan 27, 2025

View reviewed changes

drupol added 2 commits January 27, 2025 19:41

build-support: add addBinToPathHook hook

87521c5

build-support: add writableTmpDirAsHomeHook hook

f811073

drupol force-pushed the push-zyrxtzozkqnt branch from 881d72c to f811073 Compare January 27, 2025 18:42

drupol merged commit 613933c into NixOS:master Jan 30, 2025
25 of 28 checks passed

drupol deleted the push-zyrxtzozkqnt branch January 30, 2025 17:34

GaetanLepage mentioned this pull request Jan 30, 2025

xonsh.xontribs.xontrib-*: init at various #354733

Open

13 tasks

drupol mentioned this pull request Jan 31, 2025

treewide: use writableTmpDirAsHomeHook #378110

Merged

13 tasks

SuperSandro2000 reviewed Jan 31, 2025

View reviewed changes

SamLukeYes mentioned this pull request Feb 7, 2025

[24.11] xonsh: 0.18.4 -> 0.19.1 #379875

Merged

13 tasks

drupol added the backport release-24.11 Backport PR automatically label Feb 7, 2025

SamLukeYes mentioned this pull request Feb 7, 2025

xonsh: test failures prevent installation #379630

Closed

3 tasks

drupol mentioned this pull request Feb 7, 2025

[Backport release-24.11] addBinToPathHook and writableTmpDirAsHomeHook hooks #380071

Merged

13 tasks

GaetanLepage mentioned this pull request Feb 8, 2025

python312Packages.wandb: 0.19.5 -> 0.19.6 #380225

Merged

13 tasks

bjornfor reviewed Feb 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build-support: add new hooks #370869

build-support: add new hooks #370869

drupol commented Jan 4, 2025 •

edited

Loading

nixos-discourse commented Jan 4, 2025

xiaoxiangmoe commented Jan 13, 2025 •

edited by drupol

Loading

drupol commented Jan 13, 2025

hsjobeki left a comment

drupol commented Jan 14, 2025 •

edited

Loading

drupol commented Jan 26, 2025

K900 commented Jan 26, 2025

drupol commented Jan 26, 2025 •

edited

Loading

K900 Jan 27, 2025

drupol Jan 27, 2025

SuperSandro2000 Jan 31, 2025

GaetanLepage Jan 31, 2025

drupol Jan 31, 2025

SuperSandro2000 Jan 31, 2025

drupol Jan 31, 2025

SuperSandro2000 Jan 31, 2025

drupol Jan 31, 2025

nixpkgs-ci bot commented Feb 7, 2025

bjornfor Feb 8, 2025

bjornfor Feb 8, 2025

build-support: add new hooks #370869

build-support: add new hooks #370869

Conversation

drupol commented Jan 4, 2025 • edited Loading

Things done

nixos-discourse commented Jan 4, 2025

xiaoxiangmoe commented Jan 13, 2025 • edited by drupol Loading

drupol commented Jan 13, 2025

hsjobeki left a comment

Choose a reason for hiding this comment

drupol commented Jan 14, 2025 • edited Loading

drupol commented Jan 26, 2025

K900 commented Jan 26, 2025

drupol commented Jan 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixpkgs-ci bot commented Feb 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drupol commented Jan 4, 2025 •

edited

Loading

xiaoxiangmoe commented Jan 13, 2025 •

edited by drupol

Loading

drupol commented Jan 14, 2025 •

edited

Loading

drupol commented Jan 26, 2025 •

edited

Loading